Skip to content

Qualcomm AI Engine Direct - test framework refactor - #22072

Open
haowhsu-quic wants to merge 1 commit into
pytorch:mainfrom
CodeLinaro:dev_ci_3
Open

Qualcomm AI Engine Direct - test framework refactor#22072
haowhsu-quic wants to merge 1 commit into
pytorch:mainfrom
CodeLinaro:dev_ci_3

Conversation

@haowhsu-quic

@haowhsu-quic haowhsu-quic commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

Co-author: @winskuo-quic

Summary

  • extend op / feature test for htp arch (v69~v81)
  • extend op / feature test for lpai / gpu backend

Test plan

pytest backends/qualcomm/tests/rework/lpai/feature/v6/test.py --device f3c0531 --soc_model SM8850 --build_folder ./build-android/ --backend lpai
pytest backends/qualcomm/tests/rework/lpai/op/v6/test.py
pytest backends/qualcomm/tests/rework/gpu/feature/test.py --device f3c0531 --soc_model SM8650 --build_folder ./build-android/ --backend gpu
ytest backends/qualcomm/tests/rework/gpu/op/test.py --device f3c0531 --soc_model SM8650 --build_folder ./build-android/ --backend gpu

- extend op / feature test for htp arch (v69~v81)
- extend op / feature test for lpai / gpu backend
@pytorch-bot

pytorch-bot Bot commented Aug 24, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/22072

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 5191f6b with merge base b8b5e0f (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 24, 2026
@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@psiddh

psiddh commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

cc @mohankumarkumar for extend op / feature test for lpai / gpu backend

@psiddh

psiddh commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

@claude review this PR and check for any potential regressions etc

@claude

claude Bot commented Aug 28, 2026

Copy link
Copy Markdown

Claude finished @psiddh's task in 2m 43s —— View job


PR Review: Qualcomm test framework refactor

  • Gather context (diff vs merge base b8b5e0f)
  • Review non-test source changes (regression surface)
  • Review test framework changes
  • Post review feedback

I focused on regression risk. Of the 31 files / ~11k lines, all but ~7 files are additive test code (new htp/op/v69…v81, gpu, lpai test suites) that can't regress production paths. The real surface is the 7 non-test source files. Overall this looks safe to land; a few points worth confirming below.


Non-test source changes — assessment

builders/op_batch_norm.py — adds aten._native_batch_norm_legit_functional.default to the target list. Purely additive; extends coverage to the functional variant. No regression.

qnn_preprocess.py:201 — adds node.op == "call_function" guard before inserting a delegate mapping entry. This is a defensive tightening (skips non-call nodes like get_attr/placeholder that happened to carry a debug handle + single tensor name). Correct and low-risk.

utils/utils.pygenerate_gpu_compiler_spec gains a performance_mode param, inserted as the first positional arg with a default. Any existing callers using positional args would break, but all in-repo callers use keywords, and the default (kGpuPerfHintHigh) preserves prior behavior. Fine.

⚠️ _passes/lpai_partition_fallback_support.py:336PassResult(graph_module, True)
Previously the pass returned PassResult(graph_module, bool(unsupported_nodes)) (modified only when it actually inserted fallback QDQ). It now always reports modified=True, even when unsupported_nodes is empty and the graph is untouched. Depending on how the PassManager consumes the modified flag (e.g. re-running fixed-point passes or recompiling), this could cause unnecessary recompilation. Not a correctness bug, but worth a one-line justification in the PR description or a comment — the old value was intentional signal. Fix this →

handle_back_to_back_nodes(..., unsupported_nodes) — now receives the unsupported_nodes set and excludes them from the input_call_func_nodes list before the assert all(... in dq_ops) check. This fixes a latent assertion failure: an unsupported (fallback) node feeding another fallback node is legitimately not wrapped in a DQ, so the old code would have tripped the assert. Good fix.

✅ Pass-ordering swap (qnn_lpai_pass_manager.py + qnn_pass_manager.py)
LpaiPartitionFallbackSupport now runs after ResolveDebugHandle (deps [TagQuantIO, ResolveDebugHandle]), and _validate_edge_passes is rewritten to assert passes[-2] is ResolveDebugHandle / passes[-1] is LpaiPartitionFallbackSupport. The LPAI override no longer calls super()._validate_edge_passes() — correct, since the parent asserts ResolveDebugHandle is last, which is now false for LPAI. The docstring/assert message on line 62 is slightly misleading though ("ResolveDebugHandle is the last edge pass before LpaiPartitionFallbackSupport" — it's the second-to-last), minor wording nit.

⚠️ quantizer/annotators/lpai_rules.py — removed annotators (behavior change)
This removes several LPAI annotators: instance_norm, rsqrt, sort, scatter, expand_as, and drops the SharedQuantizationSpec output for topk/sort GetItem (now None). These ops are still annotated in htp_rules.py, so HTP is unaffected. For LPAI, these ops will now go unquantized and rely on LpaiPartitionFallbackSupport to fall back — which aligns with the fallback machinery being hardened above. Confirm this is the intent (drop LPAI quant support for these ops in favor of fallback), since it's a real behavioral change for any LPAI model using them. The RmsNorm change to guard node.args[2] (optional weight) is a clean correctness fix.


Test framework changes

  • conftest.py: QnnConfig(compile_only=True)QnnConfig(enable_x86_64=True), and the emulator-mode check switched from build_folder == "build-x86" to enable_x86_64. Both fields exist on QnnConfig (export_utils.py:110,117), and invoke_remote/push(inputs=...) signatures line up (export_utils.py:366). The new per-backend validate() (LPAI allows leftover quantize ops in the plan) is a reasonable relaxation.
  • test_qnn_delegate.py: new test_qnn_backend_dump_intermediate_outputs_conv_relu + ConvRelu model — additive.
  • feature.py: large but mechanical refactor — per-backend backend_compile_specs dicts, get_quantizer() helper (returns None for GPU), and TensorDump switched from an event-count assertion to a cosine-similarity numeric-gap comparison. Consistent throughout.

Minor

  • conftest.py type hint inputs: Tuple[torch.Tensor] = None should be Optional[Tuple[...]]; harmless.
  • Docstring wording nit on qnn_lpai_pass_manager.py:62 (see above).

Summary

No blocking regressions found. The two things I'd want the author to confirm explicitly are: (1) the unconditional PassResult(..., True) and (2) the intentional removal of the LPAI annotators (relying on fallback instead). HTP paths are unaffected by the annotator removal. The changes to the fallback pass and pass ordering are internally consistent and the handle_back_to_back_nodes fix is a genuine improvement.

(Review only — no code changes made. I could not run the QNN test suite here as it requires QNN SDK + a physical device.)
· branch dev_ci_3

@psiddh

psiddh commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Can you also resolve conflicts please ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants